Papers with matching pipeline
Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that large language models are contaminated with data from pretraining and finetuning tasks. |
| Approach: | They perform extensive analysis on the factors that affect model memorization and generalization, such as model size, problem difficulty, and question length. |
| Outcome: | The results show that models perform better on the subset of the benchmarks where similar solutions are seen during training. |